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Chnptor 1.7 
RECALL PERFORMANCE OF SDI PROFILES 



Assessment or the purcenta;-e Rucail porrormaiice of prof j les 
in an SDI service is undoubtedly difficult and time-consumiiur, 
and more, perhaps, tlian any other .li^jures derived from the ' 
system, the Rocall fifejures are sensitive to tlie methods 
used to obtain them. 

The general method used in the SDI Investigation was to 

send to users a list of a sample of the documents Input 

to a given matching run, asking Jthem to mark these documents 

which each would expect his profile to select. The marked 

documents were then compared with those which had been 

selected by his profile and thus the percentage Recall 

established. 

There are several major drawbacks to this method:- 

1) The number of documents relevant to the particular 
user and contained in the list is likely to be very 
small, resulting in large variations in the figures 
derived. At the same time the total number of ^i terns 
m the list may still be too large for the user to 
scan quickly and make consistent relevance assess- 

III k'-v w * 

2) The composition of a sample- list of doeumonts based 
on a week's or a fortnight ts input to the system 
will Itself vary very considerably and give uneven 
results. 
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3) The us'-^r will seldom inako the rolcvanco assess- 
ment directly on the basis of his interests as 
stated in the profile but on a looser basis 
depending on which articles strike him as of 
Lnccrcst at the time of the assessment. 

Use of a large list of documents covering a larger time- 
span may, to some extent, meet problems 1 and 2 above but 
at the same time create others. 

Only in special circumstances is it possible to ask users 
to check a large list item by item. In most cases tiie 
larger List needs to be classified to allow tlie user to 
find the small number of items on subjects of potential 
relevance to him. This In itself confines tiie possible 
choice of items to those in sections of the list which he 
can recognise from the classification system as likely 
to contain relevant articles. This being so it might be 
supposed that, depending on his particular interests and 
the type of classification used, his selection of items 
might be biased towards liis chief, perhaps easily class- 
ified and therefore re trievable in teres ts • 

In the early stages of the Inves tigo t Lon the main purpose 
of the lists sent^ out to users was less to establish a 
figure for the Recall performance jf the individual profil 
or the system as a whole, but more to discover, for the 
purpose of profile modification, wliat defects cxJr^ted in 
the profiles. Owing to the fact that major testing and^^ 
modification was still being done to profiles even after 
.the start of the weekly service this continued to be the 
maxn use made of the lists for some two months after the 
start of the operational weekly service. 

Subject, lists 

In addition to its index terms or descriptors, each 
document input to the system was assigned a set of up to 
three subject codes selected from the list in Figure h. 
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It should perhaps be emphasized that the list of subject 
and the groupings shown are not to be considered as a 
strict classification. They were designed very quickly 
as a means of roughly sub-dividing the subject area 
covered by the investigation. To a large extent the 
groupings put together subjects which tended to be grouped 
together in user»s profiles • However some of the groups 
are ^miscellaneous « categories, in which a nujnber of 
fairly unconnected subjects have been put for convenience 
simply to keep down the number of groups. 

Using these codes for sorting, the computer was able to 
produce listings of the documents input to a given veek^s 
run, in the broad subject groups. 

The interests of each user were similarly classified using 
the same broad subject classification codes. As many as 
four codes per user could bo used for this purpose. 

When it was wished to obtain Recall data for any particular 
week the ^Bulletin listing* was obtained from the computer 
on offset litho masters-, and copies produced of lists for 
each particular classification code. Users wore then sent 
copies of the lists for the appropriate subject codes with 
a covering letter asking them to mark relevant documents. 

It may be imagined that the task of duplicating, selecting 
and despatching unique combinations of lists for each user 
was no small one, particularly when the total sample of 
users or a large proportion of them wore Involved. However 
this was accepted initially to obtain the necessary data 
for profile analysis and modification. The wooks for which 
this data was collected were 01^* and 023 I.e. the fourth 
and thirteenth week of operational service since this 
period, started at week Oil. 

For the 0-]h week lis.ts the users were asked to mark relevant 
Items but at week 023 they were asked to distinguish between 
the highly relevant (ri ) and the partially relevant (R2) 
documents. The Recall figures derived were, for week 01^1, 
52 percent and for week 023, 58 percent (Recall for RI 
documents) and ^5 percent (Recall of R1/R2 documents). 

This tends to show that, as might be supposed, users 
mark more documents when thoy have a two-category mark:ing 
system rather than a single cat(jgory one, since border- 
line items which miglit be rejected In the latter system 
could be legitimately included as R2 in the former. 
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As soon as the major work of profile testing and modification 
had been completed it was decided to abandon the system of 
sending the subject Bulletin listings which required so 
much effort to produce, select and distribute. Instead it 
was decided to adopt a system of random listings in which 
a sample of the documents input to the particular week's 
run were printed out, duplicated and despatched for users 
to mark relevant items. 

The first sets of random lists covered documents input to 
the matching run for week 03 1 and three sheets wore des- 
patched to each user. On the basis of the returns the 
Recall was calculated to be 39 percent for R1 docuriients 
and 21 percent for all documents marked (i.e. HI and R2). 

However, in view of the small number of sheets sent many 
users found none of the documents to be relevant. A second 
sample listing comprising six sheets was therefore sent to 
another set of users. This latter sample gave a Recall 
figure of ^3 percent (Rl documents) and 2^ percent (R1/R2) 
documents ) • 

Further random listings were sent to users for the weeks 
050 and 056. In the latter case both random listings and 
bulletin listings were sent for purposes of comparision. 
Prom this it would appear that the Recall figures derived 
from marking of random lists giVes figures tliat are similar 
to those derived from bulletin markings assujuing that the 
sample size is not too small. 

The data derived from the various lists are set out in 
Figures 1 for bulletin listings and in Figure 2 for * 
random lists.. Data are also given for the subset of those 
profiles in the sample which remained unmodified through- 
out the investigation. In each case the figures given 
for the size of sample indicate the number of returns 
actually contributing to the figures. 

From the data in the figures it appears that the overall 
Recall performance of the system was at a fairly high 
level at the beginning of the service .i.e. for wecjks Ol'l 
and 023, dropped fairly sharply for the wenk.s 03 1 nnd O^O 
and rose again to close to its earl.ler level at weok 
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FIGURE 1. 



Recall performance of SDI profiles based on Bulletin listings 
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FIGURE 2 



Recall performance of SDI profiles based on random lists 
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FIGURE 3. 



Comparison of 023 and 056 recall using the same sampl e of users 
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Figure h 

BULLETIN SUBJECT HEADINGS AS USED FOR GENERAL CLASSIFICATION 
OF DOCUMENTS IN THE SDI INVESTIGATION 

10 Geophysics, astrophysics, astronomy, radio astronomy 
1^ Plasmas, ionization and discharges in gases 

16. Electric and magnetic fields, particle* optics , ion sources, 
accelerators,, electron guns 

18 Vacuum technology 

20 Solid state physics, crystal structure, electron states, 
acoustic, thermal, electrical properties and effects 

30 Superconductivity, ciyogenics 

32 Magnetism, magnetic materials and properties 

3^ Dielectric, materials and properties, ferroelectric! fcy 
and piezoelectricity 

^0 Optical properties of materi .Is, luminescence, 
fluorescence 

h2 Quantum electronics, masers, lasers, holography 

50 Semiconductor materials and devices, crystal growth, 
microelectronics 

60 Electron tubes, cathode ray tubes, thermionic tubes, 
pJzotomultipliers , conductors , inductors , resistors , 
capacitors , and switches 

6k Circuit theory, network analysis and synthesis 

66 Electronic circuits, amplifiers, modulators, oscillators 
logic circuits, pulse circuits, power supply circuits 
frequency dividers and multipliers 

68 Reliability, quality control , testing 

70 Telecommunications , radio , television, information and 
communication theory, signal processing 

75 Radar 

76 A»ntennas and propagation 



17-8 



77 Sonics and ultrasonics 

78 Mi c r owave te chno 1 ogy 

80 Electric machines, power conversion 

81 Aerospace facilities and techniques, space communication 

83 Direct energy conversion ^md energy storage 

85 Instarumontation and measurement, biomedical engineering, 
telemetry 

88 Particle and radiation measurement 

90 Control theory and components, switching theory, 
artificial intelligence, cybernetics 

96 Computer technology and applications 
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Chapter 18 



ASSESSMENT OF THE RECALL PERFORMANCE OF THE SDI 
SYSTEM IN RELATION TO PARTICULARLY VALUABLE ARTICLES 



In t roduc t i o n 

During the SDI Investiation it was desired to measure the 
Recall and Precision performemce of the system. or these 
Recall was, of course, the most difficult to assess* • 

The normal method was to present the user with a listing of 
a sample of the documents input to the system and to ask him 
to mark any item(s) which he considered should be selected 
by his profile. The marked items were then compared with 
those items actually notified to him and Recall was calcu- 
lated as the ratio of items marked and notified to total 
documents marked. " 

There is, of course, nothing unusual in this method. The 
point is that figures derived in this way are likely to 
underestimate the Recall performance rather than to over- 
state it. The user was free to mark items of passing interest 
or ones with no direct relationship to the subjects included 
in his profile. 

From these and other considerations it may be supposed that 
the actual Recall figures of the system based on those items 
which fall directly within the stated field of the profile 
could well be higher than the figures indicate. In particular 
the Recall performance for the important articles central to 
the user's subject field could well be considerably higher 
than for a wider sample which includes items of fringe interest ♦ 

A small study was undertaken to test this hypothesis. 
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Method 



A letter. Appendix 1 , was sent to each participant in the 
SDI Investigation asking him for details of any papers 
published in the previous six months which he had found 
particularly valuable. To avoid any direct connection with 
the SDI Investigation the letter was signed by Mr. T M 
Aitchinson, Manager, Information Research, INSPEC: (it should 
perhaps be pointed out that since the beginning of the 
Investigation all direct communications with the users had 
been signed by the author as Manager, SDI Investigation). 
The working of the letter was intended to eliminate bias 
towards SDI notifications and as far as possible to prevent 
users being influenced in any way by the SDI service. These 
considerations naturally meant that we could not ask users 
TO restrict themselves to English-language material or to 
the subject area of their SDI profiles. 

The return form left space for three articles since only the 
really important items we required and in fact the possi- 
bility of even fewer items was suggested by asking for 'NIL' 
returns. However users who wished to include more than three 
articles were permitted to do so. 

^"'^^^^'^"^^ participants- were sent the questionnaire 
and 219 replies had been received by the time it was desired 
to examine the results. The accompanying letter made a repjy 
a matter of choice so that the low percentage return (under 
50 percent) is not surprising, particularly since the question- 
naire was sent out at the height of the summer holiday period. 
Sonir^I considered likely that these particular results 

would be seriously affected by a small percentage return. 

As was to be expected only a proportion of the items cited 
had appeared among those input to the SDI system. The 
coverage of the Investigation extended only to English language 
articles in a number of chosen periodicals. Naturally some 
of the Items cited had appeared in journals other than these, 
and yet other items were from strictly speaking, non-pericdicaJ 
I't^l^tV conference proceedings. In addition, though some 
attempt was made m the covering letter to restrict items to 
those published within the period of operation of the SDI 
service, some of the papers cited had been published before the 
start of the service in November I968. For these and other 

^rth^sm S actually been input 

to the SDI Service and could therefore be used in this study. 
The number of individual returns contributing to this total 
w^a.s 1 /do • 

In using the remaining items it was realised thai the validity 
of the results would depend on the extent to which the choice 
Of Items cited was biased towards those actually notified by the 
oDI Service. 
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The most serious case would bo that in which the participant 
had intentionally restricted- the scope ot his selection to 
such articles. was, however, possible to calculate the 

maximum extent of this bias and to make allowance for it by- 
excluding any return comprised solely of items which had 
been included in the particular user's SDI notifications. 

There- is, however, one other type of bias which could affect 
the results. It may be supposed that for any person 
regularly receiving an SDI service many of the documents known 
to him will be thos notified by the service. The subset of 
such documents selected by him as particularly valuable is 
therefore likely to be biased towards SDI-notified documents. 

Little direct evidence could be derived from the sample to 
determine the effect of this second possible bias. It was 
clear from the date already listed before the corresponding 
SDI notifications were sent. These items formed a clearly 
unbiased sample, but one which was small, comprising only 
17 articles; 

In order to obtain a larger but similarly unbiased sample 
we took some of the cited items which had not been input to 
the system for various reasons. This sample included 
articles from journals not scanned for the service, articles 
from issues which had not been received originally as well as 
from issues predating the start of the service. Copies of 
these articles were obtained and passed through to the in- 
dexers m the usual manner as part of the regular input. 
The mdexers were, therefore, unaware that these items were 
in any way special and there could, therefore, be no question 
of special treatment for these articles. 

In all, some 37 articles were obtained and indexed in this 
way. A larger sample would have been preferred but the 
effort required to locate, obtain, photocopy and index the 
documents made this difficult. 

¥ith the 286 documents which had been input to the system 

o^^S J 37 articles gave a total sLple 

01 323 documents. ^ 

l^.l^l ""ir °l f't documents it was possible to find 

nan^ b'' notified to the particular partici- 

f^ t^e ^?'>e ;S *»^^-ecord of his notification! However, 
maLSL If th^ ^ ^"i'^ practical way was to do a manual * 
matching of the index terms assigned to them against the 
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This method was somewhat laborious but avoided the confusion 
that might have been caused if the documents had been input 
to the system normally and had appeared in the system normally 
and had appeared in the weekly notifications. 

In examining the profile to discover whether they would have 
selected the documents, it became clear that some of the 
documents bore little direct roJ.ation to the user's stated 
information requirements used as the basis for his SDI profile. 
This was not surprising since, in our efforts to obtain an 
unbiased sample of documents we had avoided any indication 
that the items cited should be restricted to any given field. 
This was not too serious since profiles in the Investigation 
had always attempted to cover the full interests of partici- 
pants rather than a portion only. 

However, It was apparent that some items were quite outside 
the scope of the profiles and some thought was given to re- 
moving them from the sample. This idea was abandoned owing 
to the difficulty of deciding unequivocably whether a particu- 
lar article could properly be considered to be within the scope 
of an often fairly broadly defined Statement of Information 
Requirements, and it was decided to include all items and to 
accept that this would result in a somewhat lower Recall per- 
formamce figure. 

Results 

The results of the study are summarized in Table 1. As ^ 
can be seen, of the sample of 286 documents 223 were selected 
by the SDI Service giving a Recall figure of 78 percent (aver- 
age of ratios). 

To assess the extent of bias caused by any participant 
intentionally restricting his choice only to items included 
in his SDI- notifications the figures were recalculated to 
exclude any returns where all items had teen included in 
the SDI Service and had been notified to that usero As these 
returns, by definition, produced figures of 100 percent Recall 
it was to be expected that their exclusion would considerably 
reduce the overall Recall figure. 

However, it can be seen that the figures arc very little lower, 
ie 7^ percent (average of numbers) and 72 percent (average of 
ratios). 

Using the 5^ documents comprising the 37 specially-indexed 
documents and the .17 items included in the returns before they 
were notified to the users, it was possible to establish a 
Recall figure for the system on the basis of a sample which was 
clearly free of bias towards SDI-notified documents. 
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The figure of 6l percent (average of numbers) and 65 per- 
cent (average of ratios) can be taken as the "worst base" 
estimate for the Recall performance • As stated above a 
number of the documents in the sample lay outside the scope 
Qf the profiles and could reasonably have been excluded from 
the sample with the result of increasing the calculated Recall 
for the remaining documents. The other factor tending to a 
lower Recall performance is that, at the time those documents 
were indexed, a study was being carried out of the effect on 
system performance of reducing the number of terms allocated 
in document indexing. As can be seen from Table 1 the reduced 
indexing resulted in a considerable fall in Recall performance 
and must, undoubtedly, have reduced the Recall for this 
particular sample. The end two columns of Table 1 show the 
nearest comparable Recall figures for documents in general 
calculated for both highly relevant (ri) documents alone and 
for highly relevant and partially relevant documents combin'ed 
(RI/2). It will be seen that in the period of both normal 
and reduced indexing the Recall performance for the particular 
valuable documents is considerably higher than for documents 
in general. 

Cone lusions 

It appears clear from the results that the Recall performance 
of the SDI system is considerably higher for those items 
which recipients would consider most valuable, than for items 
in general. The figure of 75 percent obtained in this study 
could well be higher if items outside the scope of the given 
profiles were excluded from the sample. 

It appears likely that a reduction in Recall performance 
affects the highly relevant documents proportionately less 
than the less relevant ones. 
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All 

documents 








RECALL 


RECALL 


Sample 


Number of 
Documents 


Number 
Notified 


Average of Average of 
Numbers ratios 


RI RI/2 


Original 
286 


286 


223 


78 76 


58 


Reduced 










125 


125 


92 


7^ 72 


58 k5 


^Unbiased 


t 








sample 


5k 


33 


61 65 


h3 2k 



I 
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Chapter 1<j 
SUBJECTIVE ASSESSMENT- OF SDI SYSTEM 



RECALL PERFORMANCE 



Introduction 



Throughout the SDI Investigation major effort was devoted 
to .easur.ng the performance of the system. Howeverlt 
thl iLl *^;"^s/ealised that in any operational service 
^s the Le^f r ^^"^ important measure of performance 
IS the -user's degree of satisfaction with the' service. 



Method 



o? ^:^"^l^?"^f*^°"naa.re sent to participants at the end 

of^.he Investigation an attempt was made to discover the 
user's assessment of the service they had received and 
the extent to which they considered It had be successful 
xn pa.cka.ng up the relevant items in their field. 

Two questions gave " us information on the user's assessment 
of the Recall performance of the service. ?hese weJe 

1. Hoy many English-language periodical articles 

va^uaMe """"^ ^^^^ particularly 

valuable xn your work? v^j.cix.i.y 

^' sLl^lTe? ^""^ ""^""^ "'^'"^ notified to you by the SDI 

4> t . _ ii'jwcvv.i u&at)j.e repJLxGs wore* -roroT^rr^H 

from almost 3OO people. wcj c rcccxvcd 
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Results 



^?ied w^Li ^i^^" by inddyidual participants 

SSlftl^ V^^^^f ° *° 3°°- '^^^ results aSe 

tabulated xn Figure 1 and show the Recall figures for 

es^ablishl^nis^^'H^ "ser groups-Universitief^ Govi^L^ent 
establism.ents, Industrial firms and Group profiles. 

^umberror^'d^^'^ "P'*^ ^^^^ ^^^^^ variation in 

percenta^J Reo"T?"^^ ^^^^"^ ^" individual returns, the 
agrees vKv c?e«ii ^^^ calculated as average of numbers 
?Swos. ^l^^^ly with that calculated as average of 

It will be seen that the Recall figure for Universltv users 

SmaV? "^^r '^r'^ caution in vif^ of ?L 

small number of people in this group. 

Figure 1 

Subjective assessment of SDI Svsten, R.^ all Perforn,.^. 



Type of 
user 


No of 
returns 


Total particu- 
larly valuable 
articles 


Total 
notified 
by SDI 


Reca 
Av. or 

Nos . 


11 fo 

Av. of 

ratios 


UNIVERSITY 


76 


2023 


1367 


68 


68 


GOVERNMENT 


93 


'2163 


■ 1264 


58 


57 


INDUSTRY 


91 


3053 


2024 


63 


65 


GROUPS 


22 


1 131 


602 


53 


53 


TOTAL 


282 


8370 


5257 


63 


62 
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Chapter 20 



RELATION BETWEEN PRECISION PERFORMANCR OF THE SYSTEM 
AND DOCUMENT INDEXINPr 



For any information retrieval system the performance must 
be dependent to some extent on the indexinfj. The most 
easily isolated feature of the indexing is tlio average 
number of terms assi{,-ned to each document that is added 
to the system. 

As the number of index terms per document decreases, it 
may be assumed that the smaller number of terms will tend 
to cover the major rather than the minor concepts in tlie 
documents: it follows that documents retrieved by these 
terms have a greater probability of being relevant. From 
a different standpoint, if the number of- document index 
terms is reduced, the number of possible retrieval 'hooks' 
IS reduced, the number of documents pulled out of the file 
IS less, giving a probability of reduced Recall and there- 
fore of increased Precision. 

It was expected therefore that an inverse relationship 
would be found betwecsn the avei'age nimiber of ijidcx terms 
assigned per document and the Precision performance of 
the system. A grapli was plotted oj- this relationship ajid 
the result is shown in Figure 1 . TJ»e result Ls not as 
clear as one would wish, the scatter of precision values 
for the sane number of index terms being very largo. 
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In view of this it vas decided to look at one of the 

assumptions underlying the supposed relationship between 
index terms and Precision. 

It is assujned that a) a reduction in the number of index 
terms will cause a reduction in the number of documents 
selected by the user«s profile, and b) the smaller nuirtber 
of documents will contain a hltshor proportion of I'elevant 
ones. Only if a) is true can b) be true. 

A plot was therefore made of the relationship between 
avera{^e number of index terms and the avera^^e number of 
notifications sent to each user. The result Is shown in 
Fi^re 2: the points relate to 'i-week periods, the number 
a{^ainst each point indicatin^j the first week of each 
period. Again, liowever, it can be seen that a clear 
relationship cannot be derived and it would' appear that 
a third factor is involved. Examination of the data in 
Pigrure 3 indicates that this may be the number of documents 
input to the system in each of the four-week periods. 

Figure shows a plot of total documents input to each 
four-week period against average number of notifications. 
It can be seen that the points for periods 11, 15 and 19 
are isolatt^d and these relate to periods wliere the number 
of documents input was relatively low. 

Finally the data in Fi^^ure 1 vas again plotted for tlie 
four- weekly periods. As can be seen from Figure 5 the 
result is a fairly satisfying curve when the precision 
figures used are calculated as average of numbers but in 
Figure 6 it can be seen that averages of percentages give 
one or two anomalous points. 

Conclusions 

It may be concluded from the results that, in tlie system 
used for the SDI investigation: 

1) As expected, the numl)er of notifications produced 
for a user in any run is directly related to tha 
number of documents input to that run. 

2) The number of notifications Is directly related to 
the average number of undex terms app I Led to each 
document in the run. The equation for the Lino 
drawn in Figure 2 which has becvn somewhat arbitrajvLI 
chosen as passing tlirough the orlgLn J s y = 6x 

5 



20-2 



'j) Th(i Procislon porrornianco i.s indiroctly related to 
tli(! avora/:o number of? Index tornis apj)!.. od to each 
document in the part j cul.-ir run. The equation for 
the line drawn in the {jraph (Kifjure 5) is y = h x + 79. 



This indicates that the Precision porfoi-mance of the system 
xs surprisingly little affected by the average number of 
index terms assigned to each document and that a figure of 
approximately 805S Precision is the highest attainable by 
means of more restricted j.ndexing. 

At the same time the number of notifications per user is 
much more affected by the average number of index terms. 
Since a considerable drop in number of notifications is 
not accompanied by any cottmiensurate increase in the per- 
centage of relevant notifications it must follow tliat the 
percentage Recall performance suffers seriously. At the 
same time it would also appear that in tliis system high 
Recall was more readily obtainable with good Precision 
than High Precision with good Recall. 
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Chapter 21 
USER ASSESSMENT' OF THE SDI SERVinR 



?^ + problems in the SDI Investigation was 

that of monitoring the performance of more than 500 
profiles, diagnosing problems and acting quickly to 
prevent any user becoming disenchanted with partici- 
pation in the Investigation. The problem conunon to all 
SDI systems is that the overall performance of the 
onJr^b.^^f T "^^^^"^S Jot the individual user who sees 
only the performance of his own profile. The chief- 
virtue of the SDI service in meeting the specific in- 
dividual needs of subscribers becomes its chief disadvan- 
litL't^^"" b^dly, since the failure is felt 

directly, personally and repeatedly week after week unless 
something is done to improve matters. unless 

The weekly service to participants in the Investigation 

deS^^ble'^o^H"'"''"" '^^^ ^* realised tha^^Jt ^as 

thJ ^! discover quickly which recipients were finding 

the service unsatisfactory so that the limited sta?f e?forf 
could be devoted to the most serious cases. After eiJht 
week* service therefore, each participant was sent a 

aspL'^:"S:'?:unf r ser:ic%'Sd Shich 

Tories w^^.r i satisfactory. The assessment cate- 

gories were Very Poor, Poor, Fair, Good and Verv Good ?Ho 
returns are sumraarized in T^ble 1. 

In the final questionnaire at the end of the Investigation 

of this second questionnaire are shovn in Table 2. 

It can be seen that the overall assessment of the service 
had improved with approximately 72 percent rating the 
service Good or Very Good compared Sith 51 pJrceSt in the 
previous questionnaire. Less^han five percent considered 
ea^lieT"' ^ -npared w?th twelvr^^r:::? 
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However, not all users who replied to the first question- 
naire replied to the second and vice versa* Tables 3 and 
4 give the results for the two questionnaires for the 
people who replied to both* These results therefore allow 
a direct comparison* 

It can be seen from these figures that the proportion" of 
people rating the service Good or Very Good had risen to 
72 percent compared with 5^ percent in the earlier question- 
naire and only five percent thought it Poor or Very Poor 
compared with 11 percent in the earlier questionnaire* These 
figures agree very well with those obtained from Tables 1 
and 2 and indicate that the sample of users replying to the 
second questionnaire was not significantly different in kind 
from that replying to the first* 

It may be concluded that during the course of the Investi- 
gation the overall degree of user satisfaction with the service 
improved* However, it is of some interest that this increase 
in satisfaction was not common to all users* Table 5 shows 
the number of people whose stated degree of satisfaction 
differed between the first and second questionnaire* It can 
be seen that although 38 percent (IIO) of the users rated 
the service better in the second questionnaire, the degree 
of satisfaction of 10 percent (29) of the users had decreased* 

Assessment compared with subjective recall perfonneaice 

In the final questionnaire, as well as giving an overall 
assessment of the SDI service, users were asked to state how 
many of the relevant articles in their field had been notified 
to them by the SDI service* It was, therefore, possible to 
compare this subjective Recall performance assessment with the 
degree of satisfaction with the service* 

The results are shown in Table 6 for 272 users* It can be 
seen that the Subjective Recall performance figure (72 per- 
cent) for those people who considered the service Good or Very 
Good was .higher on average than that {k5 percent) for those 
who considered it Fair* This, in turn, was higher than the 
figure of 3k percent for those who rated the service as Poor* 

This is, of course, what one would expect but it does serve 
to show axi agreement between these two subjective judgments • 
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Chapter 22 

RELATIONSHIP BETWEEN QUALITY OF USER^S INITIAL STATEMENT 



OF INFORMATION REQUIREMENTS AND THE SUBSEQUEWl^ PERFORMANCE 



OF HIS PROFILE 



With increasing experience of profile compilation and 
performance testing, a feeling grew that certain types of 
user statements were more likely to produce satisfactory 
profiles than others. Such a feeling might merely reflect 
a greater degree of cpnfidence on the part of the SDI 
staff dealing with certain types of requirements and it 
might not follow that the resulting profile gave greater 
satisfaction to the particular user. 

A study was therefore made to discover whether particular 
statements considered likely to give good performance 
actually did in practice perform better than others. 

Method 

A sample of 80 users was selected and an examination made 
of the original statements used for profile compilation. 
Based on this examination an assessment was made of the 
quality of the statement as Good, Poor or Fair i.e. a 
prediction of the performance of the resulting profile. 
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The factors user in assessing the statement vere:- 
POOR CHARACTERISTICS 

1 • Statement 

a) Lack of detail 

Examples of statements lacking detail are 
•I am interested in nuclear instrumentation' or 
'Please inform me of articles on airborne radar 
equipment'. Each, is sufficient to define the 
general area of interest but gives insufficient 
information for compiling a profile to select 
articles in the particular part of the field 
that IS likely to be of interest to the user. 

^) Lack of precision 

Lack of precision generally arises from failure 
of the user to realise that u greater logical 
precision than customary is needed in stating 
his requirements for the purposes of a search 
^rf; ; "^^V^ ^* common to receive statements 
of the form 'I am generally interested in semi- 
conductors of the III-V group but particularly 
Ga?, InP and InSb'. Without specifying those 
particular aspects of vhe III-V group of compounds 
xn general which are of interest, the profile can 

o^ ^„°?\^° ir^"" °^ ^^I-V compounds 

or just the three specifically-mentioned. 

The problem is not always easily resolved even 
by reference to the user since he has often not 
consciously formulated the principles on which 
he would select material himself. 

2. 3 itb.1ect Field 

a) DiCficulty of coding . 

Some subjects have a much more settled and agreed 
vocabulary than others and coding of search profiles 
IS therefore easier for these subject areas than 
for others where the vocabulary is less defined. 
Examples of areas where difficulty in coding was 
found included control, information theorjr Jnd 
man-machme systems. 
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^) Fringe interests 



A problem with any SDI system is how to decide which 
user interests can be covered by the subject coverage 
of the data base, in many cases an individual tg 
interests will fall across a number of subject areas 
some of Which will be fully covered by the particula^ 
data base, some partly and some not at all. For such 
fringe areas the performance (particularly Recall) of 
the profile and therefore the users assessment are 
likely to be poor. 



GOOD CHARACTERISTins 

^o^! ^""J the reverse of the characteristics enumerated 
above. Examples of statements with good characteristic 
each category can be seen from Table 1. 

1 . Statement 

a) Plenty of detail 

b) High precision 

2. Subject Field 
a) Ease of coding 



FAIR - CHARACTERISTins 

Statements which could not be categorized according to any 
of the above were put in the Fair category. ^ 

^o YnVk^'t t"" ""^^ ^^'^^ "^"y statements were assigned 

to 2 or 3 categories. In those cases where there was a 
mixture of good and poor characteristics, prediction o? 
c:^e~! '"^^^ ^ ^^"^^^ balancing ourof^thf 

'^L^^'^T^ performance against which the prediction was 
TeTo2tnal% ^^'^"^ °" Collection of doc^ents. 

(lll^u ^^^^^^ also available for later , runs 

oTm\ experimental service runs 001-00^, 005-006, and 

modlf?. ."^"'^ ^''^^^ ^^"^ °^ profiles whi^h had undergone 
modifications as a result of profile analysis or user 

iSeir^^iS^naf be- significantly differ^nj from 

l^ZliZ^f T compiled from the statements. These 

experimental service performance figures were in fact used 
for comparison with the user rating? of profile peJ^or^Jnce 
(obtained by questionnaires sent out aroSnd ru^ oISIHSs 

c:r?::"nderr^ ^ r^^"^^ °^ perfor^fni; S^res 

corresponded to user ratings. 
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In order to have a single overall figure for profile perfor- 
mance the product Recall X Precision was used, ie a profile 
operating at 60 percent Recall and 80 percent Precision would 
have a performance figure of 0.48 (0*6 x 0.8) • A major 
disadvantage of this method is that it assumes the equivalence 
of Recall and Precision, eg profiles operating at one hundred 
percent Recall and 20 percent Precision, or 20 percent Recall 
and one hundred percent Precision both have the same performance 
figure but are obviously not equally satisfactory* However for 
the purposes of ranking profiles this method was considered 
convenient. 

RESULTS AND DISCUSSION 

The complete details of the eighty profiles are shown in Table 1. 
Table 2 includes the.. main results: with the profiles listed in 
order of the Test Collection performance figure. 

Copies of the statements for the sample and other profiles are 
given in Appendix 22A. 

the relationships of the following were investigated: 

(1) Prediction of performance and Test Collection 
performance 

(2) User rating and the 001-014 experimental service 
performance 

(3) Prediction of performance and the 001-014 
experimental seirvice performance 

In each case the average performajice figure was calculated 
for the profiles in each performance category (see Table 3) 
and plotted against the performance category (see Figure .1 
curves 1, 2 and 3). 

In case 2, the User ratings, VP and P , and , G ajid VG, were 
combined in order to make curve 2 moxe directly comparable 
with the other two. 

It should be noted that profiles with performance figures 
of O were ignored for purposes of computing averages as 
they axe quite anomalous and usually arise from too little 
data. 

CONCLUSIONS 

(1) In general the performance predicted on the basis 
of the .statement of information requirements was 
reasonably reflected by the results obtained on 
the test collection performance (cux*ve I) but much 
less so by the results obtained in the experi- 
mental service (curve 3) when further user/system 
interaction had taken place 

(2) The user's rating of the sei^vice reflecte^l the 
performance of the searvice he received 
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Longer-term effects 



However it is possible that this higher performance and 
satisfaction is a short-teim effect based on highly specific 
short-term requirements and that over the longer-term the 
more general statement may .with the effect of interaction and 
feedback perform as well or better. 

To study this an examination was. made of the overall per- 
formance figures for the 12 months service period and of 
the user's satisfaction with the service as stated in the 
second questionnaire at the end of this time. 

Table 4 shows for each of the profiles the user's assessment 
of the service as stated in the second service questionnaire 
and the overall R1 and RI/2 Precision performance for the 12 
months service • 



In figure 2 are shown the average Precision figures for 
each of the groups of profiles derived from statements graded 
as Poor, Fair and Good respectively. This would appear to 
show that at least in the case of the Good statements the 
effect of an initial statement on subsequent performance is 
lasting and not removed by subsequent interaction with the 
user and by profile modification. 

However the sample is not of s'u'ch a nature to allow firm 
conclusions to be drawn. 

If one looks at the relationship between the predicted 
performance and the user's assessment of performance at the 
time of the first and second service questionnaire a member 
of points emerge. 

Firstly it can be seen from figure that the prediction is 
not very accurate or at least does not coincide clearly 
with the user assessment since in only 18 (375^) profiles 
is there agreement, counting Good and Very Good together. 
However in most cases the prediction errs on the conservative 
side, since 39 (80^) of profiles are assessed the same as 
the prediction or higher and only 9 (20^) are assessed lower 
than the predic tion . 

Taking the profiles in the groups predicted 'Poor', 'Fair' 
and 'Good' respectively, of the 11 profiles in the 'Poor' 
group, 3 (27^) arc assessed as Poor, while 8 (73?^) are 
assessed as Fair or Good. Of the 17 'Fair' predictions ' 
only 3 (18%) are assessed as Fair while 13 (765^) are assessed 

?^^^\^^ ^^^^ Gk)od. In the case of the 20 'Good' predictions 
12 (60%) are assessed as Good or Very Good while 8 {kO%) are 
assessed as only Fair. 

It would appear from this that the predictions are not a 
very good guide to performance as interpreted by the user, 
though there appears a greater probability that a profile 
predicted to perform badly will in fact do so than those 
which are predicted to perform v/ell. 
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However, since 64 percent of the profiles predicted to be 
Poor in fact get. a Good or Very Good rating compared with 
76 percent for those assessed as fair and 60 percent for 
those assessed as Good, it may be that factors likely to 
contribute to poor performance are more easily distinguished 
than those which contribute to good performance. 

For the second questionnaire assessments the rating of the 
*Poor* predicted profiles is again Gk)od or Very Good ?.n 7 
cases (649&) , while the figure for »Fair» predicted profiles 
has risen to ^k (82%) and for the »Good« to I8 (90%). 

An examination of the 9 Profiles which are still rated Poor 
or Fair at the time of the second questionnaire shows no clear 
pattern in the statements from which they were compiled, 
except that they covered subjects which lay somewhat on the 
fringe of the subject coverage of the service. These were 
^nuclear magnetic resonance*, *man-machine systems* and 
environmental testing. 

CONCLUSIONS 

The evidence for believing that recognisable characteristics 
of statements of . information requests will predictably affect 
the perfofTnance of a profile is not as clear as one might 
suppose. The- reason for this may be that the differences 
between statements obtained in the investigation were 
relatively small owing to the fact that users tended to 
follow closely the sample statement provided to them. 

From the point of view of the performance figures themselves, 
profiles predicted to perform well or badly do tend to show 
different performances but this difference tends to become 
less with time, probably because interaction with the user 
compensates for initial defects. 

The relationship between the performance predicted for a 
profile and the user assessment of its performance is not 
clear. In general the group of profiles predicted to perform 
badly are assessed lower than the group of profiles predicted 
to perform well . However a large number of px'of iles predicted 
to perform badly are assessed as Good by the user and vice 
versa. Profiles covering subjects on the fringe of the subject 
coverage of the sei^vice feature among those which perform 
badly as expected. At the same time not all profiles which 
are on the fringe of the field do in fact perform badly*. 
The x'eason may be the manner in which the users interests in 
the field are made up - in some cases being more closely 
identified with the aspects dealt with in journals covered by 
the service. 
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FIGURE 3. COMPARISON OT PREDICTED PERFORMANCE WITH USER 
ASSESSMENT IN FIRST QUESTIONNAIRE 

(sample of 48 USERS) 



^"-•^^^ USER 

^**>v^SSESSMENT 

PREDICTED'^'^v^ 
PERFORMANCE ^^^-^ 


POOR 

(including 
Tory poor) 


FAIR 


GOOD 


VERY GOOD 




3 (o%) 


1 (2%) 


7 (15?6) 




Fair 


1 (2?6) 


3 (6%) 


12 (15%) 


1 {2%) 


Good 




8 (17J6) 


7 (15?^) 


5 (1096) 



FIGURE 4. COMPARISaN OF PREDICTED PERFORMANCE WITH USER 
ASSESSMENT IN SECOND QUESTIONNAIRE (SAMPLE OF 

48 USiSRS) 



^-^^ USER 

v^ASSESSMENT 

PREDICTE^\. 
PERFORMANCE 
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(including 
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FAIR 
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Poor 


1 (296) 


3 {6%) 


5 (1096) 


2 (496) 


Pair 




3 (69^) 


12 (259^) 


2 (4%) 


Good 

' ' — ^ 




2 (4?^) 


13 (2796) 


5 (1096) 
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FIGURE 2. PERFORMANCE FIGURES FOR PROFILES DERIVED FROM 
STATEMENTS GRADED POOR, FAIR AND GOOD 



PREDICTION 


NO, OF PROFILES 


MEAN ANNUAL 
OVERALL PRECISION 
R1 Rl/2 


Poor 


17 


27 


66 


Fair 


26 


29 


63 


Good 


31 


3h 


75 
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xiioi^ uvr/KAl^l^ ri<JLUXC)XUl^ kUliiftiHEHVti tOH WEEKS 011-050 

and USER SATISPACTION ASSESSMENT IN QUESTIONNAIRE 
FOR SAMPLE SET OP USERS 



PROFILE 
NO* 


ASSESS- 
ncjH i at 
QUESTION- 
NAIRE 2 


OVERALL 
PRECISION 
FOR YEAR 


PROFILE - 
NO. 


ASSESS- 
MENT at 
XJUESTION- 


OVERALL 
PRECISIOl 
FOR TEAR 






2 


R1 RI/2 
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TABLE k (c 



ontinued) 



PROFILE 
NO. 


ASSESS- 
MENT 

QUESTION- 
NAIR 


OVERALL 
PRECISION 
FOR YEAR 


PROFILE 
NO. 


ASSESS- 
MENT 

QUESTION- 
NAIR 


OVERALL 
PRECISION 
FOR YEAR 


- 




2 


R1 R1/2 
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R1 R1/2 ~ 


•1 
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209 


6 


8 67 
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P 


7 44 
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13 40 
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11 51 
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78 93 
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57 82. 




242 
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36 52 
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18 28 




246 
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26 84 
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19 42 
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19 69 
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Chapter 23 



EFFECT OF PROFILE ANALYSIS AND MODIFICATIONS ON 
USERtS SATISFACTION WITH SERVICE 



It might be assumed that time and effort spent on analysis 
and modification of profiles would result in an improvement 
in performance. 

An examination was made to discover the extent to which this 
was true • 

Method 

Using the data presented in Appendix 9B, a table (Figure 1.) 
was compiled categorizing users according to their assessment 
of the service in the first and second questionnaires. For 
each category the average time spent' per profile in analysis 
and modification was worked out. 

Figure 2 shows the improvement for all profiles comparing 
the first questionnaire assessment with that for the second 
questionnaire. Thus an improvement of +1 includes all 
profiles improving from Good to Very Good, Fair to Good, 
Poor to Fair and Very Poor to Poor. Users who gave a half way 
assessment e.g. f/G are excluded. The net improvement is 
plotted against the average time spent per profile in analysis 
and modification. 

Since only one profile is represented in the figures for +3, 
-2 and -1 the points for these on the graph should be 
discounted. The remaining points cannot be claimed to show 
a pattern. 

In figures 3 > and 5 the figures for each profile are 
shown in six categorie? according to whether the assessment 
in the first questionnaire was Very Poor, Poor, Fair, 
Fairly Good, Good or Very Good. 
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I 



Again no curve could reasonably be drawn* 
Conclusion 

Prom an examination of the results ot this study there seems 
no reason to believe that there is any direct relationship 
between the amoimt of time spent in profile analysis and 
modificatioti and improvement in user satisfaction. There 
are perhaps several reasons why this might be so. 

1) It is difficult to determine how much of the time 
spent is concerned actually in improving the 
profile £ind how much in collection and analysis 
of data to decide what modification is needed. 
For complete profiles the time spent in euialysing 
large numbers of relevance assessments can be 
considerable and the result of much work may be 

a very minor chemge to the profile. 

2) Inherently unsatisfactory profiles may take up 
considerable analysis and modification time without 
*he basic fault being removed. 
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» 

1 

f 


2nd Q 


No. of profiles 


Total time 
in mins 


Average time 
per profile 
in mins 


•t 

/ VG 


• 

VG 


12 


935 


77.9 
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89.2 


ft 
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*1 
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61.9 
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P 

IT 
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VP 


VP 
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/I 
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0 


r / v 


VG 


1 


25 










25.0 


p/g 


G 


o 


155 










77.5 


p/g 
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1 
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0 



Fi^re 1. User satisfaction with SDI Service at beginning 

and end of veekly service, and time spent in profile 
analysis and modification. 



23-3 









"|y."(hiiii'~ 


■f 8 


1 




I<I0 


4 i 


21 






♦ 1 








9 


m 




1\ 


- 1 


21 






<- z 


I 






- 3 


1 


to 





23-^ 



CM 



a 

CD 
P (D 
HH 

CO P 
H» CD 
CO CO 
(0 

SCO 
CO 

pi 5 

<D 

3 » 
O c+ 

pi 

O 

H> H> 

O CO 
P U 
cf- M 



O 



CO 
(D 
4 
<{ 

O 
<D 

<D 
H 

c+ 

O 

O 
c+ 

B 

0) 

CO 



o 

H) 
H» 
H 
<D 



3f 



o 



»^ O 
0) 



CO 



o 
c 



< 



5. ^ 



4^ 
O' 
O 



H 
o 

H> 
H» 
H 
<D 
CO 

O 

H 
p. 

H 

CO 
CO 
(D 
CO 
CO 
<D 
Pi 

P) 
CO 



CP 



O <1 



X 



H 
O- 



^^ ro 
S o 

(D 



B 

CO 



V>5. 
O 



o 



wen 



CO 
CO 

CO 
CO 

B 



<D 
O 
O 

pi 

0) 
CO 
c+ 
H» 
O 

g 

K 



o 

H) 
H* 
H 
(D 
CO 

O 
K 
H» 

Jtl 
H» 

ts 

P 
H 
H 
^ 

P 
CO 
CO 

<l> 

CO 

00 

<l> 
Pi 

p 

CO 



\J1 

o:- 

o 



ERIC 



23-5 



ERLC 



cm 



s ^ 

0) 
P 0) 
H H 

CO 

CO 
CO CO 

B CO 

Pi B 

(D 
c+ 



3 
o 
Pi 

H» O 
p. 



O 
c+ 
O 



CO 



CO 
<D 
4 

O 
<D 

<D 
H 

H» 
O 

O 



B 

CO 
<D 

o 

H) 
H* 
H 
(1> 



H3 

5 

0) 
CO 



r 

C) 



O. 



o 

H) 
H» 
H 
(D 
CO 

O 
H 



P3 
00 
00 
<D 
CO 
CD 
O 
Q* 

P 
(0 



H* O 
0) 

B 
CO 



o 
o 



23-6 



< 



71 



5x 



<4 



00 
00 

CO 
0) 

5 

0) 



0) 

o 
o 
y 



CO 
c+ 

O 



H 
CD 



4 
O 
H) 
H» 
H 
(D 
00 

O 
H 
H» 
7^ 
H» 

P 
H 
H 
^ 

P 
CO 
CO 
<D 
CO 
CO 
(D 
Pi 

P 
CO 



CO p 
H' CD 
CO CO 
O 

§CD 
CO 

0) 

9 y 

O c+ 

H- O 
H>H> 

O CO 
P U 
ci- M 

O Q) 
» <t> 
• K 
<4 
H* 
O 
<D 

<D 
H 

c+ 
H* 
O 

c+ 
O 



B 

O 

0) 

<l> 
o 

H> 
H« 
H 
<D 



»-3 
0) 

S 

00 



^d 

o 

H> 
H' 
H 
<D 
CO 

O 

H- 

H' 
tJ 
P5 
H 
H 
^ 

CO 
CO 
<D 
CO 
CO 
(D 
Pi 

CO 



1-3 
H- 
B 

0) 

H- 

5 

CO 



to 

8- 



X 



> 

CO 
CO 

CO 
CO 



C/3 
<T> 
O 
O 



5 O 



0) 



0) 
O 



0) 



o 



(D 
CO 

O 

H- 

}^ 

H 
H 



CO* 

CO 

(D 

CO 

CO 

<D 

CO , 



ERLC 



23-7 



Chapter 2h 

EFFECT OF NUMBER OF DOCUMENT NOTIFICATIONS RECEIVED 
BY USER ON- HIS RELEVANCE ASSESSMENT 

It has been suggested that there may be a tendency for users 
to vary their relevance assessments depending on the number 
of document notifications they receive. Thus if a user 
receives a large amount of documents he is likely to be more 
selective in his assessment than if he receives a smaller 
number and will tend to mark fewer relevant, with a con- 
sequent tendency to give lower Precision figures. 

To determine whether this very plausible hypothesis could be 
shown to have any basis in fact an examination was made of 
the data obtained during the investigation. 

Figure 1 gives for each of weeks OII-O5O, the average number 
dWo° *° users and the Precision performance 

RV2 for both average of numbers and average of percentages. 
In Figure 2 the number of notifications are plotted against 
Precision (average of numbers) and Figure 3 shows the same 
information but using average of ratios instead. In neither 
case would there appear to be any relationship between the 
average number of notifications received and the Precision 
perf ormemce • 

CONCLUSION 

Data available from the SDI service do not appear to support 
the suggestion that users tend to mark a small proportion of 
documents relevant when they are presented with larger numbers 
of notifications. 

However in view of the considerable number of uncontrolled 
variables including, most noticeably, the completely different 
sample of source journals and documents represented in each 
week's run, it is in no way possible to draw any stronger con- 
clusions. 
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PXUUKJS 1. AVERAGE NUMBER OP NOTT FICATIOKS AMD PRECTSTQW PTOTTPIi^g 
FOR WEEKS Oll-Otn 
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'eek No. 



Average notifications 
per user 



Figure 1. (Contd) 
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Chapter 25 



COMPARISON O F RELEVANCE ASSESSMENTS BASED ON THR 
FULL DOCUMENT W ITH THOSE BASED ON CARD NOTIFIC ATTONS 
GIVING V ARYING AMOUNTS OF DOCUMENT INFORMATION 

In the vast majority of SdI systems the user receives as the 
product of the search, not the selected documents themselves, 
but some substitute (surrogate), commonly including the author, 
txtle, source reference, index terms and possibly an abstract 
ol the document. The object is. to give him sufficient in- 
formation in a convenient form to allow him to decide whether 
the document is relevant and to chose whether he wishes to 
obtain the full text. 



The question arises as to the information which should b« 
provided in the document surrogate to aiiow the user to make 
a reliable judgraenj; of the relevance of the original to his 
needs . 



METHOD 



Four different types of notification cards were produced 
containing different amounts of information. The basic 
information was in each case the same i.e. author, title 
of document and journal reference. In the case of Group II 
notifications this was the only information supplied. Groups 
I, III and IV contained additional information as follows:- 

Group I Basic information plus index terms assigned by 
the SDI indexers. (This was the infoiTnation 
normally supplied on SDI notification cards). 

Group II Basic information only 

Group III Basic information plus abstract, 

Group IV Basic information but with title augmented 

to make it more explanatory where necessary; 
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Normally all documents received were numbered serially after 
indexing but for this study they were numbered before' being 
passed to the indexers . The numbering was carried out as 
follows. The first four documents were numbered 062 001 - 
OOH, the second four 062011-014 etc. Thus each document was 
assigned to one of the four classes depending on the terminal 
digit. This ensured that documents arriving together e.g. 
articles in the same journal were not all grouped together and - 
gave a reasonably comparable sample in each group, while allowing 
documents to be numbered and processed immediately on receipt as 
was essential for the currency of the SDI Service. At the same 
time the coincidence of the Group number and the terminal digit 
simplified considerably the later procedures and analysis. 

After numbering, the documents were indexed as usual and in 
addition an augmented title was written where necessary for 
documents assigned to Group IV (terminal digit k) , and for those 
documents in Group III (terminal digit 3) which did not have 
a usable abstract, an abstract was written. The size of 
abstract was limited bv the space available on the card. A 
maximum of approximately 80 words was possible and some abstracts 
had to be shortened. 

All the data except the abstract was input to the computer by 
means of punched paper tape as normal, and thus appeared on the 
notification cards produced from the tapes via offset litho masters. 
All that was necessary was to add the abstracts for the Group III 
documents to the offset litho master^ Thus all users received a 
set of notif: cations containing a mixed set of Group I, 11, m 
or IV notif i- ation cards. No note explaining the presence of 
abstracts on some cards or the absence of index terms on others was 
sent to users and it is interesting that very little comment was 
received from them. 

Examples of the various notifications are shown in Figures 1 - k. 

Several weeks later all users who returned relevance assessments 
for these documents were sent copies of the documents and asked 
for relevance assessments based on the fulldocumeht. To save 
expense only approximately half of the documents were treated 
in this way. The sample was made by sending copies of only those 
documents having an even fifth dig5± in the serial number e.g. 
062362 thus giving a sample evenly spread throughout" the collection 
and therefore not confined to articles from a few journals. The 
covering letter sent with the sample of documents is shown in 
Appendix 10 ^ 

For each document assessed, the following information was recorded - 
user number, document number, relevance assessment based on card 
notification, and relevance assessment based on full document. 
The relevance assessments were in each case either 1, 2 or X, being 
respectively 'highly relevant', 'partially relevant' and 'not- 
relevant ' . 
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RESULTS 



The results are shown In figure 5 

It is clear from thefi'gures that the best correlation with the 
full document assessment is given by the assessment based on the 
notification which include the augmented title and by those 
which include an abstract. The worst agreement is that for 
the title only' notifications, though the range between all 
four IS not very great. Perhaps suprising is the fact that 
provision of an abstract does not appear to increas the chance 
of agreement between the card assessment and the full document 
assessment, as compared with the augmented title. 

It may be concluded that an abstract does not make a 
significant contribution in helping the user to decide 
relevance of the original, document. 
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FIGURE 1. Group I notification card - normal SDI notification 



062511? 99; 



Upp«r-atao sphere winds and their interpretation- II 
Turbulence in the lower £-region; 

D, Layzer and J. P. Bedinger; 

Planet Space Sci. Vol.17 No. 11 I89I-.I911 November 
1969; 

Ionosphere; Atmospheric Currents; 

Variations; E. Region; 



FIGURE 2. Group II notification card - title only 



062012; 99; 



Using thermostimulated exoelectron emission for 
ionizing radiation dosimetz*y; 

A.I, Beskorskii and pothers; 

lustrum. Exper. Tech. No. 1 35-38 January-February 
1969; 
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FIGURE 3. Group III notification card - abstract 



062533; 99; 

Mttasuring narrow f .m. deviation; 
R«A« Kennedy; 

Marconi Instrtua. Vol.12 No. 3 5^-55 September 

On the P.M. /a.m. Modulation Meter type TP 
2300 the lowest deviation range is 5 KHz 
full-scale. ■ Although this is adequate 
for conventional applications it does not 
provide sufficient discrimination to measure 
the very low spurious f .m. from- mobile or 
broadcast transmitters. This requirement " 
can be met by using an external voltmeter 
to measure the l.f. output of the Modulation 

Meter which enables deviations as low as 
10 Hz to be measured. 



FIGURE k. Group IV notification card - augmented titl< 



06255'*; 99; 

An inexpensive multichannel scaler with channel 
widths of less than 1 microsec: uses a time- 
to-pulse-height converter; 

E.C. Silverberg; ' 

Rev. Sci. Instrum. Vol.^^O No. 11 1530-150i* 
November I969; 
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FIGURE 5m Relevance assessments of various card notif icationi 
compared vith those based on full document* 
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Chapter 26 

Effect of Incl uding author names aa search terms in SDI 
Profiles . 

The statement of Information Requirements form se?t to 
each user asked for details of the subjects on which he 
would wish to receive information. No attempt was made 
to discover the names of authors whose work would be likely 
to be of interest. Thus except for one or two users who 
volunteered some information on this point, SDI profiles 
throughout the Investigation did not contain author names 
and searched solely on subject matter. 

It was however intended to ask users for names of suitable 
authors and to include these in the profiles at some con- 
venient time during the Investigation. For this reason the 
authors of documents were included as searchable elements 
m the document record during the early stages of the In- 
vestigation. (It should perhaps be pointed out that to 
^ i''^^.^^^''^^ °" authors to be made the names had to be 
J"^ ilJf descriptors along with the subject descriptors 
J'^o? H ^°'r""«nt record). Until the author names could be 
included m profiles this part of the record was not usable 
and when a way was being sought of reducing the punching 
load from week 026, the author descriptors were obvious 
candidates for dropping until the temporary flood of innnt 
had been dealt with. The inflow of documents however re- 
Zl^'^t^K^ ^ ^i'"^ and it was not ■ 

?ecoid ev!n J^^^^^^^-. reinstating the author elements of the 
m^ ty. ^"-^^^ °^ authors had now been obtained 

from -Che users for inclusion in profiles. It will be 
realised too that the punching effect required to add an 

wirifi^sei?'^ n? *° °^ profiles 

was in Itself not readily available at this time. Apart 

Sat owinr^n*^^" P^^^^-^-^S effort it must be remembered 
that owing to the computer program difficulties, profile 
modification was at no time a straightforward matter and 
there was a natural reluctance to face the disruption of 
the service that could result from profile modification on 
this scale. 
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Nevertheless the facility of searching by author names 
IS iroportant and it was not satisCagtory to carry out 
the Investigation without at least ^examining the effect 
Of xncluding this option. A small investigation was 
therefore planned to discover how the inclusion of author 
names m SDI profiles would affect the performAice of the 
SDI servxce. 



Methods 



Given the constraints mentioned above i.e. the neeH to 
keep the punching load to a minimum and to disturb "the 
adopted! ^^^^^^ possible the following^ methods were 

1) The author descriptors were added to the documents 
as surname without initials e.g. FREEMAN rather 
than FREEMAN F J. The difference may appear slight 
but m fact reduced to less than half the time- 
taken and the error rate in punching since errors 
occur mainly with the spacing and puncuation of 
initials. 

2) Instead of adding the authors to the individual 

^SutioJ^n^n?^^-^*'-''^'-^ "^"^^^ (25) of special 

author' profiles were constructed made up solely 
of the author names for the profiles as a whole. 
In these again the surnames only were used. The 
o^+^f these special profiles avoided disturbance 
of the individual profiles and kept the matching 
^he'^nLrT ^'^^ [^-"^^ors investigation' separaWrom 
^c^?iny? available for separate 

ar?icSs ^^''''^^S" °^ profiles was to select 

■t-^ J-iiitiaxs. me particular users interes + PH 

m these authors could then be looked ud in I • ^ . 

the appropriate notifications sent to tL^ ? 
assessment. ctuions sent to them for relevance 

?5 course, a laborious method but it was con 
deficiencies in^i, required to overcome the 

Organisation 

A letter (Appendix 10A82)- was sent to all users invitine- tb«™ 
o? in?^Jrst!'"''^ °' i^ely^fo^b:" 
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Although a reply to the letter was left optional most 
participants did respond and few gave less than ten 
neunes . 

It was intended to compile the author profiles and to 
run them for ten consecutive weeks and to use the out- 
put for this particular study. However considerable 
difficulties were encountered in getting the profiles 
on file and the period of time that each profile was on 
file varied. 

At the end of the period the items produced by the author 
profiles were sent for relevance assessment to the 
appropriate participants. 

In view of the fact that each of the author names given 
by a participant was on file for a different length of 

^^nn ^5%''^^"^*! """"^^ "^'^^ *° show what propor- 

tion of documents retrieved might be caused by author 
names in normal circumstances. A manual search was there- 
for undertaken to assess this. 

The output for each of 2k randomly-selected profiles was 
examined over a period of ten weeks to discover w^^th^r 
euiy of the documents were written by authors named. 
The results of this are shown in Table 1. 

Results 

^* Q% i«L^^''*i:''o?o"* ' supplied names of authors, 

93 vere sent 259 documents to assess. These 
comprised 14? individual documents. 

2. Relevance assessments were received for I63 of 
the 259 documents, 93 being assessed as rele- 
vance 2. , 

3. 80 (49/0 of the 163 documents were also retrieved 

! /"^^^''^ P^o^'iles i.e. 83 were re- 

trieved only by author. Of these, 61 (749^) ^ere 
Classed as relevant (ri or R2) anA 3k \kO%) as 
highly relevant (R1). \ /"J 

k. Of the 80 documents retrieved by both subject 
and author, 79 (99/c) were rel vant and 62 (780/.) 
vere highly relevant. ^' 

5. From the results of the manual matching (Table I) 

^'^^^ °^ 971 relevant dofuients ^ 
selected by the subject profiles of 2k par^ici- 

brlutho'"^ ^^'^'^ ^-^^ been'^^Il^ecied 

.by author nsunesr 

It must be rememtered that some of the names supplied by 
participants were those of foreign authors which could not 
be expected to retrieve documents from largely English- 
language periodicals. e-i-J-o" 
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However the number of documents retrieved by subject 
from a file including foreign language articles would 
have been proportionately larger and there is no 
reason to believe that the percentage of documents 
retrieved by author names would have been greater. 



Conclusions 



«?, h""^ hesitant to draw very firm conclusions from the 
study. However, it seems clear that with such a small 
percentage of relevant documents retrieved by author 
compared with those retrieved by subject, the overall 

°^ ^'^^ ^""^ ^^^^^'^^ " unlikely to hive been 
pro?i^:s.''""°"^'^ '""^ exclusion of author nLes ^^om- 

ifsmaf? til Pf^"'^^"*^^^ °^ documents retrieved by author 
IS small the relevance of such documents on averace is 
much higher than those retrieved by subjec? ^^ 86 
per cent are relevant and 57 per cent highly Relevant 
compared with fibres of approximately 70 per cenl^d 
33 per cent for documents retrieved by subject! 
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jfile No. °^ notifications sent No. of author 

for weeks 050 - 059 Batches for 

050 051 052 053 054 055 056 057 058 059 Total 050 - o59 
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* A dAsh indicates that no assessment is available either because there was no 
latching for that week, or because assessments were not retumeS. e.g» In the 
5a8e of 079 where the profile was on a very limited topic the dashes indicate the 
•ormer. Jn 588 where the output was normally quite high, from 050-053 the user 
*ailed to return assessments* 



Chapter 27 

VARIATION OF USER SATISFACTION WITH COMPILER OF PROFILE 



All search profiles used in the Investigation were compiled 
internally by INSPEC staff. The majority of profilesTere 
compiled by two of the Indexer/Analysts who remained on Jhe 
staff throughout the Investigation. The first of these 
^Compiler No. l) vas responsible for 230 profiles and the 

m""^ ?o/°* profiles. The remainder of 

the profiles, 126 m all, were compiled by various other 
members of the staff during the course of the investigation 

?Se; do°nS fn'* """^ *° ^-other 

tney do not form a homogeneous group dince many after compil- 
ation by one person, underwent subsequent analysis and 
modification by others. 

* # 

However for the sake of identification these profiles are 
assigned to compiler No. 3. Details of the profiles assigned 
to each compiler are given in Appendix 2? A. 

It °f ^""'^ interest where profiles are compiled 

f^""^^ ^^l^i'' *° discover whether the performance of 

for Inl^^fr/^^ differs and what factors might account 

lor any difference found. 

As far as formal education is concerned Compilers 1 and 2 
may be considered generally equal in that they were both 
science graduates without directly applicable training in 
the specific field covered by the investigation i.e. 
electronics research, but they were specialists in the field 
Ox pxiysics. , 

In terms of experience of information work, CompMe- 1 had 
several years advantage over Compiler 2 who was a recent 
graduate. However it is likely that the greatest diffe.-once 
between the two was greatest in temperament and approach to 

vSlirComnSTp'^ ^ systematic analytical approach 

While- Compiler 2 took perhaps a more intuitive line. 
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A comparison was made of the profiles compiied by eacli 
compiler to discover whether the respective users were 
equally satisfied with the service received w The first 
service questionnaire returns for each Compiler are shown 
in Pigvire 1 , and those for the second questionnaire in 
Figure 2* 

The figures are shown as histograms for each Compiler in 
Figures 3-5^ For the first questionnaire returns, the 
general distribution of user assessments is very similai^ 
although the figures for each category of satisfaction differ 
in soiTcJ cases e.g. 38^ Fair for Compiler 1 against 33^ for 
Compiler 2. However it ifv not possible on these figures to 
say that there is any overall difference in user satisfaction 
for the two compilers; since the total in each case who 
express positive satisfaction (Good or Very Gk>od) is very 
similar, 53*5^ for Compiler 1 and ^k.k'ji for^ Compiler 2 with 
a higher number of Ve/.y (rood assessments in the latter case. 
This slightly greater degree of satisfaction for Compiler 2 
px^ofiles is offset by the fact that 12.5^ of Compiler 2 
users express dissatisfaction (Poor or Very Poor) as against 
8.4^ for Compiler 1 profiles. The Fair assessment is really 
too neutral to be considered as evidence on its own. 

It can only be considered from these figures that there 
appears to be no appreciable difference between the two sets 
of profiles in terms of user satisfaction. 

It must be admitted that this was a most unexpected result 
since for various reasons the Compiler 1 were thought likely 
to be superior in this respect. 

The comparison of Compiler 1 and 2 profiles with Compiler 3 
profiles is striking. The histograms show very great differ- 
ences, with CompiJ.er 3 users having the highest percentage 
of Very Good profiJ.es but the lowest percentage of Combined 
Good and Very Good profiles. It is, however, not possible 
to consider this group of profiles as homogeneous in view of 
the various people who were responsible for them. In addition, 
many of these profiles were compiled at a very early stage 
in the investigation before the Thesaurus was complete and 
were therefore in some cases deficient in some terms. 

In the second questionnaire it can be seen t?iat though satis- 
faction appears to have increased for all three groups of 
profiles, the increase in satisfaction is greater for Compiler 
1 profiles than for Compiler 2 profiles. In the former group 
the percentage of Risers exprossj.ng positive satisfactior 
(Very Good or Good) has increased by approximately 20 percent 
while for the latter group the increase is only approximately 
13 percent. 
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Increased satisfaction is however greater in the case or the 
compiler 3 profile where positive satisfaction has increased 
from H8 to 72 percent i.e. by 24 percent. 

To discover whether this improvement could be related to th*- 
times spent in profile analysis and modification by each- 
compiler, the relevant data were obtained and are shown in 
Figures 6-8. 

The average time for profile analysis and modification for 
each compiler is as follows :- 

Compiler 1 - 8k minutes 
Compiler 2-91 minutes 
Compiler 3 - lO? minutes 

It appears therefore that this in itself does not affect the 
increase in user satisfaction. 

CONCLUSIONS 

It would be unwise to attempt fi^rm conclusions but one or 
two are suggested. 

1) User satisfaction appears to be largely determined 
by factors other than the person compiling the 
profile since overall satisfaction is verv similar 
for both Compiler 1 and Compil*»r 2. ' * 

2) Nevertheless by subsequent analysis and modifica- 
tions an increase in satisfaction is attainable 
and Compiler 1 appears to have achieved a some- 
what greater increase than Compiler 2. 

3) Compiler 3 profiles initially exhibit a diff extent 
satisfaction distribution from the other profiles 
presumably reflecting a mix of profiles compiled 
at widely different times by different people. 
However it is noticeable that by the time of the 
second questionnaire after revision they exhibit 
much the same characteristics . as the other two 
groups of profiles. The greater increase in 
satisfaction may well be due to their starting 
from a lower level of satisfaction originally. 

It is perhaps surprising that the individual profile compiler 
appears to make only a relatively small impact on user satis- 
laction. Some possible reasons may be as follows :- 

l) Profile compilation, analysis nnd modification 
IS aimed primarily at achieving good performance 
m terms of the performance measures used in the 
system. 
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2) User satisfaction, though broadly related to 
measured performance of the profiles does not 
correspond directly to it in all cases. Thus it is 
possible for a compiler to achieve a higher per- 
formance in terms of greatly increased Precision 
with relatively little loss of Recall, but give 
less satisfaction* to a user who is concerned with 
high Recall almost to the exclusion of Precision* 

3) The user's satisfaction with the performance of 
his profile is almost entirely determined by the 
extent to which he has clearly stated his require- 
ments, how easily these can be coded, how consist- 
ently he stands by these stated/requirements in 
assessing the sei*vice, and, not least, by his 
temperament. Given all these major factors in 
determining user satisfaction, the scope for the 
profile compiler may be very restricted. 
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FIGURE 1. 

USER SATISFACTION AT FIRST QUESTIONNAIRE AND 
PROFILE COMPILER 
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Chapter 28 



METHODS OF USER INTERACTION TN . 

PROFILE COMPILATION 

A 

A proposal to include a study of the effect of different 
types of user interaction was made, in the course of the 
Investigation, in December I967. The proposal is given 
in full in Appendix 28A. The types of interaction 
compared were: 

(a) No interaction 

The p-rofile was compiled on the basis of the user's 
st-:.fcement of information requirements. 

(P) Profile for comments 

A draft profile wac submitted to the user for coamicnt. 
Modifications were made, on the basis of his c-laments. 
to form the final profile. 

(C) Questions' 

After the user's statement of information requirements 
had been studied and a draft profile compiled, 
questions were addressed to thr ,user wherever the-e was 
the slightest doubt as to the compiler ' s understanding 
of his requirements. The f.i.nal profile was compiled in 
the light of his reply. 
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(D) Interview 

The user was interviewed by a member of the SDI staff 
to discuss his statement of information requirements, 
Its implications, etc. The final profile was compiled 
on the basis of the statement and interview. 

( E ) As required , 

In drafting the profile, the compiler was free to have 
no interaction, send the profile for comment or submit 
questions as he considered necessary.. Because of the 
travelling time which would have been involved it was 
decided that the 'interview' option should not be 
included. 

It was originally intended to continue the comparison of 
interaction in the profile modification by dividing all the 
profiles used in the compilation study into two groups, 
the first of which (x) would have no interaction with the 
user (other than the normal relevance assessments of the 
output), and the second (y) would have any or no interaction 
as found necessary. However, because of the over-riding 
need to ensure that as many profiles as possible were satis- 
factory before the start- of the experimental which was 
expected in April 1968, it was decided to make no differen- 
tiation between X and Y profiles and to modify all profiles 
without interaction with the user. 



Selection of .groups 

To ensure that 'a number of users were available for inter- 
view at a location, the random selection of the five groups, 
A,B,C,D and E, yas made in sequences of five consecutive * 
profile numbers. 

A random selection was made, first by type of organisation 
{le academic, governmental or industriaa), then by 
organisation, and finally by users within the organisation 
{m sequences of five users). 

At this stage in the 'Investigation a considerable number 
of profiles had been compiled. This ;i.n itseJf did not 
necessitate their exclusion from tfie study since no inter- 
action had taken place and the profiles could bo considorod 
draft profiles only, so long as they had not been modi.riod ir 
response to relevance assessments. 



28-2 



However, those profiles which wore "oricinally compiled 
during the first few months of the Investigation wero 
omitted from the study, since this was a learning period 
durxng which expertise in profile compilation was built 
up within the team and, perhaps of more importance for 
the study, the working thesaurus was under its most 
intensive development. 

Similarly th^^re were a substantial number of profiJes 
whxch for a, variety of reasons were dealt with too late 
to be xncluded in the study, ie the need to compile them 
and have them performing satisfactorily at the start of 
the experimental service made it impossible to carry out 
any of the interactions with the user. The equivalent 
no interaction" profiles were also omitted. 

Impleme ntation of interaction methods 

Group B - profile s submitted for commnnt 

Since a controlled- language was used, based on a 
thesaurus to which the user did not have access it 
was realised that this method of interaction wa- 
unlikely to be very productive. The letter exp:.aining 
what was required of the user (if he wished to take part 
in the study} is given in Appendix ZSB. As may be seen 
much of it had to be devoted to an explana.tion of the 
controlled language and the profile logic. 

Group C - questions posed by letter 

For Group C profiles, the compilers were required to 
isolate all the-points in each statement of information 
requirements on which there could be any doubt as to 
the user's meaning or intention. Questions wore comoilecl 
s^it /^'if' ^^1"^^^^*^°" °n these point, aui which wSe 
?o ^^^^"^ review. The coverin,i Xetlor sent 

to users with the queries is gi ^on in Appendix 28C and an 
example of a question form in Appendix 28D. 

Group D - interview s 

^^J'lt the investment in staff time 

Un travelling etc) was so great for interviews that 11 
It essential that +he maximum information should be ' ' 
of interview. Since, for convenience, all 

of the Group D users in one location would be interviewed 

o? Til t^s%'prof •r'"''^"/^° ""^^^^'^^^ thrj^m^ilor 

wo.ni ? ! P^°^^l°«' ^'^s arrange-' that the compiler 
ZZt H J. .*^f "^'^^ queries to be answered in t)^ inter- 
view, and that the interviewer would incorporate these 
queries in preparing the- interview structure > 
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•To ensure that a consistent approach was adopted in 
preparing for the interviews, a skeletal interview 
plan was produced as shown in Appendix 28E. This 
was used in the preparatory work to form the basis 
Of the interview questionnaire, an example of which 
xs shown in Appendix 28F. 

Group E - interaction as required 

profiles in Group E were dealt with 
by one of the above methods, the only difference being 

of interaction (or none) was most appropriate. 



Performance Tests 



When the profile had been compiled it was tested against 
the test collection of documents (Test l). if the per! 

^efr-rth^t 3ati3faetiry, thi profile was 

J-eit m that form for the experimental service. 

Where the performance of the profile was considered 
inadequate, the relevance assessments were anai;Jed. 
modifications made to the profile on the basis oJ the 
imilV "^^^ modified profile again matched against 

(?esria)':eir 1°"* ^^^^ °^ second iest 

^Test la) were also considered unsatisfactory the pro- 
cedure was repeated in Test lb. ^ 

Professional effor t in comp i 1 ^i:-; nr, 

^or*an ll7tl"l^"\r^ """^'"^"^ preparation of profiles 
lor an SDI system there are a number of different pro 
fessional (as opposed to clerical) act-iv?f5ot ^ . 

The-sp inoi,,^^^ 7- J, ^J-i-iii-ai; activities required. 
These included,, for the system used in thu Ixivcstigation:- 

(1) assimilation of the information provided by the 
user m his statement of information requirements 

(2) compilation of draft pi;ofilo 

(3) interaction with the user 

(k) modification of profile in the light of the 
interaction 

For all of these activities the professional ^-r-r 

professional time was u:^od. For Group C proX: LJ thn oje- 
ments included in the Lntoraction -times cojnpri.s..d Iho " 
^nd J^P°" <iouh^^ul points, the framing of .suitable que.st.i.on.s , 
and the review of those questions before submission. 
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Results 



Porf ormanco 



In Ki/';Mr<: 1 llic juiiubor- ol' proTi Los in oach /croup which 
woro submitted to th(j Ihroo tests are shown. As can 
be seen from that f injure and in the alternative display 
in Fif^re ^4, interviewing is by far the most successful 
and no interaction the least successful in producing an 
immediately-satisfactory profile. This is confirmed in 
results for the performance in Test 1 which is tabulated 
in Figure 2. to have 3ubstantially disappeared (Figure 3) 
and to be* roughly equivalent to the use of questions or 
the 'as required' choice of interaction methods. The 
•no interaction' method continues to have the least 
satisfactory performance, although the disparity is 
considerably reduced. A surprising result, however, is 
the clearly superior performance of the profile interaction 
method. Since the difference in Test 1 between the 
profiles which were modified as a result of the partici- 
pant's comments on his profile and those which were not 
is minor ( 

s^iifi in fact favours the unmodified profiles) j 
the final test- performance might reasonably have been 
expected to equate to that of Group A, ie profiles with 
no interaction. A. possible explanation, in particular 
for the considerable superiorit3'- in Precision, is that 
the compiler, in considering further modifications after 
Test 1, felt constrained by the knowledge that the user 
had signified his acceptance of the profile in its original 
form and made only the most essential modifications to the 
profiles in that froup, whereas, for all other groups the 
tendency was to make more extensive modifications with a 
view to improving Recall at the cost of Precision. 

Professional effort 

The time spent by professional staff in conj. lation, inter- 
action and modification before Tost 1 is tab dialed for each 
group in Figure 2 and the ranking in Figure k. As expected 
the time is least f or . the non-interaction group (a) and most 
for the interview group (d) with the profile group (b) the 
•as required" group (e) and the "questions" group (c) lying 
between them, in that order. . ' 

When the time spent up to the final test is considered 
(Figure 3} the same ranking is shown but the differences 
between groups are reduced. 



Discussion and Conclusions 

The results of the study show that thoro is a dooidnd 
.advantage in interaction wlt;h tho nso.i-, s.inco Llw .i.n:M;.ial 
saving in professional offort ga.i.nod Uy hav Lnr no .i n lorac on 
is reduced by the incroasod offorL ri:qii.lrod .snl)scqi:on 1. 1 y 
and the level of perforiiianco of tJio proriJ<!S does not maJ.ch 
that of profiles compllod with user InLorac tion. 
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It is more difficult to draw gonoral conclusions on the 
nost cost/cffoctivo method of interaction because of the 
special circumstances of the SDI Investigation. The 
questi-on method of interaction wil.l be ffenorally 
applicable, .whatever the type of SDI system, but the 
results for the method in which users were asked to 
comment on their profiles are severely limited in their 
application by the fact that a controlled lan^age was 
used and a listing of the vocabulary was not available 
to the user. Again time spent on the profiles for which 
the interaction was by inten^r-Lew was especially large 
because the service was being supplied centrally to users 
scattered throughout Britain. If the service were being 
provided in one location or if the profile compilation 
wore being carried out by agents in the same location, the 
txme would be greatly reduced and would be likely to be 
only slightly greater than that for the method in wliich 
the liser is questioned by letter. 

However the change to a local system of profile compilation 
would not improve the performance of the profiles or the 
degree of interaction, wh6reas the change of an uncontrol]ed 
language would increase the oppor'tunity for the user to 
comment on the profiles. 

In general, since the difference in performance between 
dxfferent interaction groups of profiles is is small, the 
most cost effectiveness method would seem to be that in which 
the professional staff cost is least, ie submission of the 
profile for comment by the user. 
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Figure 

Ranking of Groups 
by various measures 
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RankiTig kveii Tost sp^^^ pro tncuice In pre final in anec 



i oiilv. 



Tost 1. Test 1 Test I n J-inal 

To si. 



1 (best) D ('-,5) A U-6") 1) {■-)H/r)7) A (l" :3I.") M («'3/67) 

2 E (27) 13 (i» hy) c (5:i/56) fi3 (;,'« 3'3") /e (82/59) 
» (26)\ E (a« 5'i") ^E ('19/51)'^ E (2« Zi3") (78/62) 
c (26) J c (2« 10") (^B •(^j7/56)) c (3' 08") Id (79/60)/ 

5 A (11) D (5» 31") A (/irv'<8) D (6« 38") A {75/58) 
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Chapter 29 



USE MAI)]': OF SDT NOTIFICATIONS 



Introduction 

The re^^lar feedback of information on the performance of 
the SDI Service consisted of the weekly relevance returns. 
These indicated which notifications were relevant in two 
categories: R1 - highly relevant, and R2 - partially 
relevant. Non-^relevant notifications were marked X. Such 
returns, of course, give limited information but at was 
decided to restrict the demands on participants to the 
basic minimum in order to encourage prompt and regular 
returns • 

To discover a little more of what recipients did with the 
notifications and in particular how many of the articles 
notified were read, a special study was made. 

Method 

For one week^s notifications each participant was asked to 
indicate on his relevance return whether he 

a) intended to read the particular article 
or b) would file the card for future use 
or c) would discard the reference 

Tlie work was carried out over a period of six weeks, a 
proportion of the participants in turn bein^ asked for 
this information. 

Results 

The results are sununarlzed in Table 1 

It can be seen from the Table that of the 3086 notifications 
sent, the recipients intended to read 756 {270/0) of them and 
to file a further 1^163 {h8o/o) for future reforenco. As 
expected the vast majority of those .Liitondod to l)e road wore 
RA (.highly relevant ) dociuncnts Lhoii^ih a not xusl/';n.i f ican t 
number of R2 documents were incliidod. 



29-1 



The niujoril.y oC tho LU.'iii.s niod Tor Tuturo rei-orcnco wc-o 
of course, R2 documents though it is interesting- to find 
that some 1^5 items marked as non-relevant were in fact 
filed for future, reference. Without more evidence it is 
impossible to ^a^r whether these represented useful in- 
formation- not -di^rectly relevant to the profile or whether 
cards were being filed indiscriminately! m this co^ec^ion 
It IS encourapng to find that most of the non-relevant 
Items were being discarded. -texevanr 
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Chapter 30 



Desirability of including material other than En^qi sh-lan^uarce 
periodical articles in the coverafcc of the SDI Service 



The SDI Service provided in. the Investigation was restricted 
xn coverage to English- langi.iage periodical articles* 

In the Pinal Questionnaire of the Investigation participants 
were asked to indicate how desirable they thought it that the 
SDI Service should cover other sources of information. The 
possible sources were listed as follows:- 

1 • Books 

2. Conference proceedings and papers 

3. Foreign- language pejriodical articles 
k. Manufacturers* literature 

5» Patents 

6. Reports 

7. Standards and specifications 

8. Theses and dissertations 

To each class of material participants wore asked to assigri 
the values 0, 1 or 2, having the following meanings :- 

0 - Don*t mind if omitted 

1 - Inclusion desirable , 

2 - Inclusion essential 

Results 

The results are .tabuJatcd boJow. It wi.ir bo remomboroH LIiaL 
the participants in the Investigation were, In LUo, main randomly- 
selected individuals drawn roughl.y oquaJ.ly from universities and 
colleges, government establislimen ts and Indus trxuJ firms. The 
breakdown for these three types of user is shown in the tabl<j/ 
The 'Miscellaneous' group consists of Group users and some non- 
randomly selected individuals. 
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The weighted totals are arrived at simpAy by counting 0 for 
the catecory 0 items, 1 for the category ^^ items, and 2 Tor 
the cato/^or-y 2 Uems* PVom these figures the sources may be 
ranked in Lho ordor^oP r)rf!rorenco for coverage. 



Souf^ce 

Conferences 

Foreign periodicals 

Reports 

Theses 

Books 

Patents 

Manufacturers » 1 \ terature 
Standards and Specifications 

J 



Score 

hZ2 
315 
295 
217 
^76 

162 
12l' 
10'< 



Hank 

1 

2 

3 
U 

5 
6 

7 
8 



The same rankings occur if-bnly the figures for the category 
2 items are taken, ie those which were considered essential 
for inclusion. Tliis is convenient since it avo L is any 
discussion of the relative weight to be assigned to category 1 
and category 2 items. 

It is, of course, also possible to consider the figures from 
the opposite point of view, ie which items were considered most 
expendable by the participants. If we do this we find, as 
might be expected, the reverse ordej^ of ranking from that given 
in figure 2, with one exception: foreign periodicals are 
considered more expendable than reports. 

If we compare the rankings assignod by university, government 
and industrial participants there are one or two differences 
but these consist mainly in university partici pants » placing, 
as might have been expected, a high value on thesis material, 
and industry rating patent literature rather more liighly. 

Rank 



University Government Industry 



Conferences 1 

Foreign periodical 2 

Reports l\ 

Theses , 3 

Books 5 

Patents 6 
Manufacturers* literature 8 
Standards and 

Specifications 7 



1 
9 
3 

r> 
7 

6 

8 



1 

3 
2 

7 
5 
/| 
6 

8 
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20 


128 


1 


28 
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52 
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29 


'16 


19 


53 
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0 
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17 


76 


56 


28 


7 1 


1 0 




Weighted total 


58 


123 


102 


18 


^3 


78 


23 
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3 


20 


6 


1 1 
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1 
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oft 0 


0 


62 


19 


36 


66 


72 


27 


75 


60 


^17 


Weighted total 


50 


131 


99 


^♦8 


/♦o 


97 


3'« 


55 




Industry 




















2 


7 


36 


1/4 


9 


6 


18 


3 


3 


95 


1 


32 




50 


26 


'4 2 


5/4 


26 


35 


31/1 


0 


63 


17 


38 


66 
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26 


73 


6^4 
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Weighted total 
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78 


kk 


5/« 


90 


32 
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Miscellaneous 
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8 


2 
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8 
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7 


56 


1 
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20 
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1^ 
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0 


15 


h 
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75 
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77 


•135 


977 


0 
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50 


98 
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198 


92 


2h 1 


155 


^•■55 


Weighted total 


176 


/|22 


315 


121 


162 


295 


101 


217 
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Chapter 31 

NOVKLTY O F INPOKMAT.tON PKOVXDJOJ) MV TIIK SDJ SKRVICE 
Introduction 

For any subscriber to an SDI or any other curront- 
awarenoss service, the notifications he receives Vroin 

^^^Lr'^'^'^r ""i^^ °^ information 

M^"y subscribers will regularly 
see the two, three or more Journals central to Wielr 

articlpf '■"m'^'T^^'I'^ ^«S"l^^-ly <=ontain relevant 

articles. >fost of the articles in these journals will 

Sr?nr'' "'"'"^^ *° subscriber before ' 

sVon oJ ^^^^-^^^^^r^^ hope to notify him of them. Inclu- 
^o^^-Sf T articles among his notifications will se^ve 
possibly to confirm, that the service is effective 117 
he IS unlikely to be willing to pay for a servile 'which 
?miLrT? -l^--dy knows. In 

hT«^-:i?^:;-? ^.''-r^^-^^ service is 

inf-^ion-ih-c-^^^^^^^^^^ 

An attempt was made to assess this in the SDI Investigation. 
Method 

t^giv: "if Idd"?.*f T'i""^ participant was asked 

for eIcA St?c?e ro^ f° ^''^ relevance assessment 

or eacn article notilied, an indication of whether 

Six weeks. "''^ ""^ " P°r:i.oci „r 

Results 

The resnlt.s are shown .in Table 1 
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Table 1 
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/Coo 


2-37 


217 




156 


66 


183 


8/| 


339 


75 








19'i 


311 


505 


1'|/| 


7'« 


268 


86 


'412 


81 
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/JO 


82 


30 


71 
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83 


63 


77 


7 
i 
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186 


3^9 


135 


83 


166 


89 


301 


86 




7 • 


608 


178 


23/^ 




137 


77 


205 


88 


3'i2 


83 


9 


22 




66 


105 


171 


52 


22 




82 


I't2 


82 


al 


362 


2828 


880 


1093 


1973 


65^ 


7'» 


9^0 


86 


1599 


81 



Columns 1 - 3 show for each week the number of participants 
xnvolved and the total number of notifications they 
received. Columns h - 6 show the number of those notifi- 
cations which wore highly relevant (ri) emd of secondary 
relevance (R2). Columns 7, 9 and 11 give the numbers of 
RI and R2 notifications referring to articles of which the 
receipient was not previously aware. Columns 8, 10 and 12 
show these as a percentage of the relevant notifications 
sent m each category, RI , R2 and R1+R2. 

It can be seen from the table that Ih percent of the higli 
relevance (Rl) articles notified by the SDI Service had not 
previously been seen by participants. The figure for articles 
marked R2 (partially relevant) is i>omewhat higher i.e. 86 
percent, giving an overall fitTuru for both RI 'and R2 articles 
of tJI percent. 

"Useful" Precision 

On the basis of the figines :in Lho table it i.s, of course, 
possible to arrive at. figures for tlie "UsofnJ" preo;i.sion 
performance of the system. I.e. relevant no ti C i oa Lions not 
previously seen expressed as a percentage of l.otal. not.iri.- 
cations . 



31-2 



Notifications 


Rol 


ovant 


Not Previous Iv 




isdLon 


uselux 




R1 




seon 






Precision 




in+R2 


R1 R1+R2 


R1 


R1+R2 
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Chapter 32 



ACCEPTADILITY TO USKRS OF ALTlCimATIVK FORM 
OF SDI NOTIFICATIONS 



Introduction 

In desi^nin^ the SDI SorvLce .it was decided that the notl- 
rications shouJd be sent in the Torm oV cards* Of the 
possible standard sizes, 5" x 3", 6" x 'j", and 8" x 5", .it 
was decided to adopt 6" x since the small card placed 
too tight a restriction on the amount of information that 
could be included. The 8" x 5" card was considered rather 
larger than required and such information as we had 
indicated that potential participants In the Investigation 
used this size less than the smaller cards. 

However, although cards were thought Likely to be the most 
flexible and convenient form of notification a study was 
planned as part of the Investigation to discover whether an 
alternative form of notification would be acceptable to 
recipients. 

The obvious alternative to the cards was a computcj-- 
produced listing of matching documents giving the author, 
title, citation and descriptors. 

Method 

It was originally planned to provide the alternative form 
of notification for four consecutive weeks during the middle 
of the twelve-months operational' period. However, owing to 
lengthy delays in obtaining minor adjustments to the .layout 
of the matching output, it was not possibJe to f)rovide more 
than one week's not i ficat.i ons in this form hoVora the end <>r 
this period of the Investigation in Decemht^r Tlu; 
lateness of this study iiad pcsriiaps onu ixdviuilur^-. in Uuil it 
occurred after rc^cipicnts lind been iurorim^d of the; immincMit 
changeovc^r to n cos t- recovery scM-viccr. Tliu.: there w;is nu 
.incentive? for pee^plc; tc; f:onsi(l(M- s'-rioiisly the (|iiestioti o I' 
preJerr(;cl J'or-m ol" ncj t i T i c;i t i cms since they wc^ulcl shm-tly lur 
pay.ing for the service. 
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For Uio wook jn question (Wcck nuiiibor - scries O63) 
rocipioiifs uarti sunt n printout showi.n^^ details of the 
clf.v.iiiiHMi t.s m;> tf:h i.rw7 th»; i r- profilers Lnstcjad of the set of 
<:ar<ls uot'uuilly scjnt. The a.ccoinpany Lrwv questionnaire 
(apponda X 3'-A)xsked users to indjcate their views on the 
new form of notifications as follows: 

a) Very much prefer the new form of notification 

b) Somewhat prefer the new form of notification 

c) Very much prefer the cards 

d) Somewhat prefer the cards 

,e) Have no particular preference 

The questionnaire also asked the reasons for any particular 
preference • 

Reasons for preferred form of notifications 

The SDI participants, in addition to stating their prefer- 
ence, were asked to give their reasons briefly. 

The reasons given covered various merits and defects of the 
two types of notification, e.g. readability, ease of handling, 
etc, and though these were expressed in different ways it 
was possible to group them. 

The reasons for preferring cards are summarized in Table J. 
The groupings obviously overlap since the flexibility 
associated with one record per card is closely connected with 
the argument for ease of filing. However, the headin^^^s 
service to group the reasons stated or clearly implied by 
the users. 



Hosu 1 Ls 



Of tl.« /07 participants who wore sent no tiricutions in 

' I f?""" ^"""^ ^" replies. The results arc shown 

in Table 1 . 



Table 1. User preference for card 



s vs. paper print-out 



CAI 

Very, much 
prefer 


IDS 

Somewhat. . 
prefer 


LIS! 

Very, much 
prefer 


'ING 

Somewhat 
prefer 


No 

' ■Prcfercrico 


To fcal ' 


257 
i72fo) 


63 

(1850 
■ 


5 

(Wife) 


17 
(^.750. 


17 


359 



A preference for cards was expected, but the almost 
unanimous preference did cause some surprise. A breaks 
dpvn by type of organisation is shown in Table 2 



Table 2 Preference for nards bv typ. of or?.ani t i 



Type of 
User 



Very much 
prefer 



Somewhat 
prefer 



Very much 
prefer 



Somewhat 
prcX'cr 



No 

Preference 



Table 3 



i 



Hoasons in lavour or cards Number or people 

mentioning 



FILING 

More convenient ror riling and 
later reference 

Compatible with normal filing 
system 

Assists, further sub-ciassification 
by the user 



HANDLING 

Less liable to damage or wear 
Easier to handle and read 

FLEXIBILITY 



Convenient to have one card for each 
individual paper 



OTHER REASONS 

Have become used to cards, inconvenient 
to change 

Family like to write on the dud cards 
at homo 



this factor 

2 53 
23 
29 



8 

20 



Easier to pass individual references 

to colleagues 3^* 



Ik 



Easier for taking to library to look 
up relevant articles 

Convenient for use in requesting loan 9 

RECORDING OF ADDITIONAL INFORMATION' 

Summary of the article can be written on the 
back 

There is space for more detailed abstract /| 
Additional information can be added 



10 



Table 3 (continued) 



LEGIBILITY 

Cards easier to read, better layout 
of information, upper and lower case 
easier lo read than line-printer 
output . 



Number of people 

nifjn L i on Ln/v 
_ This I'acior 



40 



DISCARDING IRRET.RVANT TTRMS 

Convenient to discard irre.levant 
material or give special attention 
to urgent items. 
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Table k 

Number of People 

Reasons for prefor ring Comnut ftr Lis tine: Mentioning 

This factor 

FILING 

Easier to file ' ^ 

Occupy less file' space since fewer 

individual pieces of paper ^ 

— -Adaptable- t of older sirorage' j 

HANDLING 

Easier to handle ^ 
FLEXIBILITY 

Esier to take to library ^ 

LEGIBILITY 

Easier to scan folder of pages 

fhan individual cards ^ 
Easier to read and understand ^ 
Easier to see at a glance 2 

RELEVANCE RETURNS 

Easier for assessment of relevance 5 
Self-copying form 
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Table <■) 

Number of People 
mentioning 

Disadvantage of eithn-r form of notification thi^s factor 

CARDS 

More easily lost 

1 

Wrong size, should be 5" x 7" ^ 

PRINTOUT 

Difficult to file 

Not so easily stored 

Sensible filing system impossible 

Filing by subject not possible 

Would involve transfer- of information 
to cards for filing 

Very inconvenient to keep 

Great deal of unused paper 

Inconvenient size, incompatible with 
other paper formats 

More difficult to read 

Difficult to scan 

More easily torn 

Irrelevant items cannot be discarded 

Individual items cannot be passed to 
members of research group -^causing 



5 



3 
1 
6 

7 
k 
1 



5 



5 



3-' -7 



CONCI'USfONS 



Jh^jr . ""T"^'' '>vorwlio.ln,.i.ncly to jiistify the original 
choxcc ol cards as the Torm of notifications. Very fow 
people (6 3?^) showed any proforence for the listimr and 
only I.Zjfo had a strong preference. Strong preference for 
^h^ tllT °" °*her hand was shown by J^i of the users. 

?^esr;;rtH r.J''" P^^f-r^-^nce appear generally agreed. 
These are that the cards provide a conveniently-handled 
xndxyxdual record for each item which can bo fLeS^ passed 
to other interested people or discarded as required. 

Some other reasons for preferring the cards m„st be treated 
xn coritext, particularly tha.t of greater legj.bil-ity. Par^lv 
thxs xs duo to the fact that the cards use u^per a^d loSer 

p?intoj;r 'on'fh^'L*"^"^^" ^"^^^ °^ computer 

printout. On the other hand the layout of the information 

on the printout could have been improved, in paJ?icuTar b? 

separating the descriptors from the JourAal rejerence? 

It must also be said that the printout was a new form imposed 

lZ±cTteT%Z^l T T some pio^ir^ 

xndicated, they had adapted to the cards and the printout 
brought problems of compatibility. printout 

Miscellaneous comments 

A number of the replies contained comments of a general 

"tendeS ^""^ ''^^'^"'^ ^^^^^ contained abstracts which 

tended to be more informative than j,ist keywords. 

The keywords section might be clearer for sortin/^ if ,1^.^ 
one line from the title. sorting if dropped 

I shall withdraw from the scheme if the new form of r.r.i---r- 
tion IS introduced. • ^ notifica- 

It is possible to include brief abstracts. 

from a centra.!, eompuLcr .store. ' ' 
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A format within 5" x 2" printed on thin paper for sticking 
to cards in a filing system would bo most welcome. 

Punched classified cards would be idca.l . 

There is a general problem that many titles of articles, 
particularly American, are not an immediate indication 
of contents. A very brief synopsis in such cases would 
. give some meaning. 

"Title of Paper" information is often insufficient to 
determine relevance of technical content. The difficulties 
or providing further information is appreciated but one 
wonders what effect this may have had on the results of 
this study. 

There would be some advantage in attempting to ^--ive 
references the classification used in Current Papers and 
Abstracts. 

I would be inclined not to use the service unless cards 
were at least available on request. 

If the references were available on IBM punched cards the 
value of the service would be increased tremendously. 

A sensible filing system for the references, eg by topic 
xs impossible with this new form of notification. The SDI 
service has been useful in providing l) an up-to-date 
appraisal of the literature, 2) a permanent record of the 
literature. Adoption of this new form of notifications 
would seriously impair the usefulness of the service to me.. 

A useful addition would be the place of origin, since the 
authors are not always known, and journals not always 
immediately available elsewhere. 

Sometime ago you were able to append short abstracts to the 
references. This increased the vaJuo of the service enor- 

"°W^;r r"^"^ worthwhile- if only, say, 3O/0 

- HQo/c of references had abstracts even if ihe cost of the 
service (to the user) went up by about ^0% _ 20</o. ■ 

The language in which the article is available wouJd be of 
interest if noted, also if available :in translation, for 
instance "m Russian, English edition available". 

The address of the authors, so that reprints of important 

Jc^.^at^'^nS?" without having to consult the 

actual publication, sivould be printed on the carcJs. 



'^i^.u^'^'S •^yf**^"' °^ presentation is one of the main benefits 
of the SDI Scheme as far as I am concerned. 

It would be helpful to have the date of printing or deJ ivory 
on the cards," 
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